Some statistical and logical considerations when rescoring tests

When tests or portions of tests are scored subjectively by raters, a rescoring will yield a change in the ratings of some examinees. In a test with a xed passing score a rescoring will result in the change of some pass/fail decisions. The number of changes depends on three things: the reliability of the rating system, the proportion of examinees that initially pass, and the policy used to incorporate the rescore into the pass/fail decision. In this study, we provide a model that facilitates the evaluation of various rescoring strategies. We consider and compare the e ciency of three rescoring strategies: (1) rescore everyone, (2) rescore failures only, and (3) rescore within some range of the passing cuto . These rescoring strategies are evaluated by direct simulation. Additionally we consider the optimal allocation of rescores where the probability someone asks Eric T. Bradlow ([email protected], (215) 898-8255 (Ph), (215) 898-2534 (Fax)) is Assistant Professor of Marketing and Statistics, Wharton School of Business, 3620 Locust Walk, University of Pennsylvania, Philadelphia, PA. Howard Wainer ([email protected]) is Principal Research Scientist, Statistics and Psychometric Research Group, Educational Testing Service, Rosedale Road MS 15-T, Princeton, NJ. The authors thank Jinming Zhang for his work on earlier versions of this research. This work grew out of a discussion of new problems facing licensing exams at the monthly COPA Research Seminar. We are grateful to the participants of that seminar for their helpful discussions. This research was supported by ETS's Research allocation to the Research Statistic Group.